Scan-skip test goes blind if the call chain grows 3 frames (#707) - #708
Scan-skip test goes blind if the call chain grows 3 frames (#707)#708philcunliffe wants to merge 4 commits into
Conversation
… MaintenanceReport symmetry, span visibility, LLP gloss PR #701 was squash-merged at head 287b67b, before the round-2 review fixes in cc82d6f were pushed, so four verified fixes never reached master. - Hoist a cheap `compactionDue` check above the `hasResettleCandidate` row scan and gate the scan on `!compactionDue`. Recognition of a foreign sorted `replace` outranks the resettle check, so the first tick after each foreign replace paid a complete single-column scan of the day purely to discard the answer. - Add `totalRebaselined` to `MaintenanceReport` beside `totalCompacted`, and let `query.js` read it instead of re-deriving the count. - Tag the enclosing `maintenance.partition` span with `rebaselined`; the `hyp_rebaselines` counter carries only the dataset, not the partition. - Give LLP 0199's bare `Extended-by: LLP 0207` the corpus's linked and glossed form. Co-Authored-By: Claude <noreply@anthropic.com>
Round-1 review of #706 approved the carry but flagged that none of the four stranded fixes was pinned by a committed test, plus two @ref nits. - Add three tests to test/core/cache-retention-maintenance.test.js, in the foreign-sorted-replace block: - `totalRebaselined === 1` on a re-baselining run, and `=== 0` once converged (pins the MaintenanceReport symmetry fix). - a capturing TracerProvider asserting the maintenance.partition span carries `rebaselined: true` (pins the span-attribute fix). - a partition already due for compaction: assert the resettle candidate's data file is read once, not twice, by spying on `fs.readFileSync` (pins the scan-skip fix). hasResettleCandidate's return value is otherwise unobservable once compactionDue is true (it's discarded via `||`), so this is the cheapest honest signal available; each new test was verified to fail when its corresponding fix is reverted. - Retarget the scan-skip gate's @ref from LLP 0207#foreign-replace to #outranks-resettle: the gloss ("recognition ... still outranks it") is that anchor's actual subject, not the recognition test's. - Drop the @ref on the span-attribute comment: #re-baseline settles the cursor-write shape, not telemetry, so citing it was close to mechanical. The prose rationale stays; it's the useful part. - Amend the PR body's claim about item 1: `needsCompaction` (pure, read-only) is now evaluated unconditionally in the re-settle path where the old `||` short-circuited past it, so more moved than "only the scan's side effect is skipped." Co-Authored-By: Claude <noreply@anthropic.com>
- test/core/cache-retention-maintenance.test.js: the resettle-scan-skip test asserted a global .parquet readFileSync count of 1 for the whole maintainCache tick, not just reads the gate governs. Any future second legitimate read elsewhere in the tick would break it with a message that misdirects the next reader. Switch to capturing a stack trace per .parquet read and asserting none pass through hasResettleCandidate, attributing each read to its caller instead of counting tick-wide. Verified both directions: passes at this head (3x, no flake), and fails with the expected message when the !compactionDue && gate in src/core/cache/maintenance.js is reverted, showing hasResettleCandidate in the offending stack. - PR body: item 2's exemplar list cited llp/0012:9, 0017:9, 0036:9, 0041:8, 0191:9 as using the linked-and-glossed Extended-by form, but only 0191 actually carries a link; the other four are gloss-only. Replaced with docs confirmed to use the linked form: llp/0106:9, 0129:9, 0158:9, 0180:9, 0182:9, 0188:9, 0190:9, 0191:9. (0201, also suggested, only has a backward "Extends" pointer, not "Extended-by", so it was excluded.) Co-Authored-By: Claude <noreply@anthropic.com>
`a partition already due for compaction skips the resettle-candidate row scan` attributes each `.parquet` read to its caller by capturing `new Error().stack` inside a `readFileSync` mock and asserting no stack passes through `hasResettleCandidate`. The discriminating frame sits at frame 8 of 10, and `Error.stackTraceLimit` defaults to 10. Three more frames anywhere between the mock and the caller (an icebird refactor, extra node:test mock internals, a wrapper in `resolver.js`) drop that frame, and a negative "no stack mentions X" assertion then passes vacuously: the test stays green even with the `!compactionDue &&` gate reverted. The `stacks.length > 0` sanity check only proved a read happened, not that the stacks were attributable, and the `new Error().stack ?? ''` fallback would store an unattributable empty string without complaint. Demonstrated by running with `Error.stackTraceLimit = 7` (standing in for the three extra frames) and the gate reverted: the old test passed. Two guards, because they cover different routes to the same blindness: 1. Raise `Error.stackTraceLimit` to 50 while the mock is installed, restoring the previous value in the existing `finally` so it is put back even if the test throws. This removes the practical hazard. 2. Assert positively that some captured stack names `compactGeneration`, the legitimate reader. It calls `scanRowsFromTable` from exactly the same depth as `hasResettleCandidate` does, so any truncation deep enough to hide the frame the negative assertion hunts for also hides this one and fails the test loudly. Asserting on `scanRowsFromTable` itself would not work: it sits one frame shallower and survives truncation that has already blinded the real check. This guard also catches the empty-string fallback. Under the same truncation with the gate still reverted, the hardened test fails; with the gate restored it passes, truncated or not. No production code changed. Co-Authored-By: Claude <noreply@anthropic.com>
Review round 1 - head
|
| step | test | truncation | gate | result |
|---|---|---|---|---|
| 0 | old (47fd4a4) |
none | reverted | fails, at async hasResettleCandidate present |
| 1 | old | stackTraceLimit = 7 via NODE_OPTIONS --import |
reverted | passes - blindness reproduced |
| 3 | new | external 7 | reverted | fails on the attribution assertion |
| 3b | new, in-test limit forced to 7 | n/a | reverted | fails on the new sanity assertion |
| 3b' | new, in-test limit forced to 7 | n/a | restored | fails loudly - guard 2 is gate-independent, as intended |
| 4 | new | none, then external 7, then real --stack-trace-limit=7 |
restored | passes in all three |
Step 0's stack is exactly 10 frames with hasResettleCandidate at frame 8, so the comment's "2 frames of headroom, 3 extra frames drops it" arithmetic is exact rather than approximate. Step 1 is the load-bearing one: the old test really does go green under truncation with the gate reverted, so the defect this PR fixes was genuine.
The scanRowsFromTable call, verified independently
Dumping the good-path stacks gives 15 frames with scanRowsFromTable at 7 and compactGeneration at 8 - exactly the depth hasResettleCandidate occupies on the bad path, with scanRowsFromTable one frame shallower in both. So the author's non-obvious call is right: asserting on scanRowsFromTable would survive truncation to 7 that has already blinded the negative check, reproducing the very blindness the fix removes. Choosing compactGeneration is load-bearing, not stylistic.
Restoration, vacuity and cost
originalStackTraceLimitis captured before thetry, the raise is inside it, and the restore is the first statement offinally, ahead of theawait fs.rm- so a failing cleanup cannot skip it. Verified empirically by injecting a forced assertion failure mid-test and probing from a later test: the limit was back to 10.node:testruns top-level tests in a file with concurrency 1, so the raised global is never visible to a sibling even while in effect, and nothing else in the file touchesError.stackTraceLimitor.stack.stacks.some(...)on an empty array isfalse, so the new assertion cannot pass vacuously; with the existingstacks.length > 0ahead of it, the?? ''empty-string hole really is closed.- No measurable cost: 5 runs each, old 23.4-26.6 ms vs new 22.7-23.5 ms. The gate means one
.parquetread per run and the full stack is 15 frames, well under 50.
Gates
npm test 3903 pass / 0 fail / 6 skipped, typecheck clean, smokes cache_lifecycle_maintenance, incremental_sink_compaction, cache_roundtrip all ok. Target file run 5 times consecutively: 38/38 each, no flake. No em dashes, no code semicolons, and the comment prose matches both guards and the measured frame depths.
Note for whoever merges: this PR is stacked on #706 and should land after it.
Root cause
test/core/cache-retention-maintenance.test.js, testa partition already due for compaction skips the resettle-candidate row scan, attributes each.parquetreadFileSyncto its caller by capturingnew Error().stackinside a mock and asserting no stack passes throughhasResettleCandidate.The discriminating frame sits at frame 8 of 10, and
Error.stackTraceLimitdefaults to 10:Two frames of headroom. If the chain between the mock and
hasResettleCandidategains 3 or more frames (an icebird refactor, extranode:testmock internals, a wrapper inresolver.js), the frame silently falls off and the negative assertionstacks.filter(s => s.includes('hasResettleCandidate'))passes vacuously - the test stays green even with the!compactionDue &&gate reverted.The existing
stacks.length > 0sanity check only proved a read happened, not that stacks were attributable, and thenew Error().stack ?? ''fallback would store an unattributable empty string without failing.The fix
Both hardenings suggested on the issue, because they cover different routes to the same blindness:
Error.stackTraceLimitto 50 while the mock is installed, restoring the previous value in the existingfinallyso it is put back even if the test throws. This removes the practical hazard outright.compactGeneration, the legitimate reader. It callsscanRowsFromTablefrom exactly the same depth ashasResettleCandidatedoes, so any truncation deep enough to hide the frame the negative assertion hunts for also hides this one, and the test fails loudly instead of going quiet. Asserting onscanRowsFromTableitself would not work: it sits one frame shallower and survives truncation that has already blinded the real check. This guard also catches the?? ''empty-string fallback.The existing comment's explanation of why a mock-based observation is the only option (
hasResettleCandidateis module-private,scanRowsFromTableis an unpatchable ESM named import) is kept, and extended to describe the two new guards.No production code changed.
Demonstration
Truncation is simulated with a preload module (
Error.stackTraceLimit = 7, standing in for three extra frames) passed viaNODE_OPTIONS="--import ...".Step 0 - the test is sensitive today at the default limit
Gate reverted (
const hasResettle = settle), no truncation:Step 1 - blindness reproduced
Gate still reverted, plus truncation. Old test:
Green with the gate reverted. Exactly the defect the issue describes.
Step 2 - hardening applied
(the diff in this PR)
Step 3 - hardened test fails under the same truncation, gate still reverted
Guard 1 (the limit raise) defeats the external truncation, so the real check sees the frame again and fires:
Step 3b - guard 2, checked independently
Simulating truncation that the raise cannot fix (in-test limit forced to 7, as if the chain had outgrown even 50), gate still reverted. The positive assertion fires rather than the test going quiet:
Step 4 - gate restored, hardened test passes
Stacking
This stacks on #706 and should merge after it. The test being hardened was added by #706 (
fix/issue-700-followup) and is not onmaster; this branch is based on #706's head47fd4a4, so the fix applies to the code it targets. The diff here is one commit touching only the test file.Gate
npm test: 3907 tests,# pass 3903,# fail 0(6 pre-existing skips)npm run typecheck: cleannpm run smoke -- cache_lifecycle_maintenance: oknpm run smoke -- incremental_sink_compaction: oknpm run smoke -- cache_roundtrip: okFixes #707